Grouped Knowledge Distillation for Deep Face Recognition

Compared with the feature-based distillation methods, logits can liberalize requirements of consistent feature dimension between teacher and student networks, while performance is deemed inferior in face recognition. One major challenge that light-weight network has difficulty fitting target due to its low model capacity, which attributed significant number identities Therefore, we seek probe extract primary knowledge related identity, discard others, make more achievable for network. Specifically, there a tail group near-zero values prediction, containing minor distillation. To provide clear perspective impact, first partition into two groups, i.e., Primary Group Secondary Group, according cumulative probability softened prediction. Then, reorganize Knowledge Distillation (KD) loss grouped three parts, Primary-KD, Secondary-KD, Binary-KD. Primary-KD refers distilling from teacher, Secondary-KD aims refine but increases distillation, Binary-KD ensures consistency distribution student. We experimentally found (1) are indispensable KD, (2) culprit restricting KD at bottleneck. propose Grouped (GKD) retains omits ultimate calculation. Extensive experimental results on popular recognition benchmarks demonstrate superiority proposed GKD over state-of-the-art methods.